This application is a continuation of (1) U.S. application Ser. No. 14/641,243 entitled Client-Server Visualization System with Hybrid Data Processing, filed Mar. 6, 2015, which claims priority to and is a continuation of (2) U.S. application Ser. No. 12/275,834 entitled Client-Server Visualization System with Hybrid Data Processing, filed Nov. 21, 2008 which issued as U.S. Pat. No. 9,019,287 on Apr. 28, 2015, which claims the benefit of priority of (3) U.S. Provisional Patent Application Ser. No. 60/989,881 filed Nov. 23, 2007 and (4) U.S. Provisional Patent Application Ser. No. 60/989,913 filed Nov. 23, 2007, the teachings of each of (1)-(4) are incorporated herein by reference in their entireties.
The invention pertains to digital data processing and, more particularly, by way of example, to the visualization of 3D and 4D image data. It has application to areas including, by way of non-limiting example, medical imaging, atmospheric studies, astrophysics, and geophysics. 3D and 4D image data is routinely acquired with computer tomographic scanners (CT), magnetic resonance imaging scanners (MRI), confocal microscopes, 3D ultrasound devices, positron emission tomographic scanners (PET) and other imaging devices. Medical imaging is just one example of a market that uses these devices. It is growing rapidly, with new CT scanners, for example, collecting ever greater amounts of data even more quickly than previous generation scanners. As this trend continues across many markets, the demand for better and faster visualization methods that allow users to interact with the image data in real-time will increase.
Standard visualization methods fall within the scope of volume rendering techniques (VRT), shaded volume rendering techniques (sVRT), maximum intensity projection (MIP), oblique slicing or multiplanar reformats (MPR), axial/sagittal and coronal slice display, and thick slices (also called slabs). In the following, these and other related techniques are collectively referred to as “volume rendering.” In medical imaging, for example, volume rendering is used to display 3D images from 3D image datasets, where a typical 3D image dataset is a large number of 2D slice images acquired by a CT or MRI scanner and stored in a data structure.
The rendition of such images can be quite compute intensive and therefore takes a long time on a standard computer, especially, when the data sets are large. Too long compute times can, for example, prevent the interactive exploration of data sets, where a user wants to change viewing parameters, such as the viewing position interactively, which requires several screen updates per second (typically 5-25 updates/second), thus requiring rendering times of fractions of a second or less per image.
Several approaches have been taken to tackle this performance problem. For example, special-purchase chips have been constructed to implement volume rendering in hardware. Another approach is to employ texture hardware built into high-end graphics workstations or graphics super-computers, such as for example Silicon Graphics Onyx computers with Infinite Reality and graphics. More recently, standard graphics boards, such as NVIDIA's Geforce and Quadro FX series, as well as AMD/ATI's respective products, are also offering the same or greater capabilities as far as programmability and texture memory access are concerned.
Typically hardware for accelerated volume rendering must be installed in the computer (e.g., workstation) that is used for data analysis. While this has the advantage of permitting ready visualization of data sets that are under analysis, it has several drawbacks. First of all, every computer which is to be used for data analysis needs to be equipped with appropriate volume-rendering hardware, as well as enough main memory to handle large data sets. Second the data sets often need to be transferred from a central store (e.g., a main enterprise server), where they are normally stored, to those local workstations prior to analysis and visualization, thus potentially causing long wait times for the user during transfer.
Several solutions have been proposed in which data processing applications running on a server are controlled from a client computer, thus, avoiding the need to equip it with the full hardware needed for image processing/visualization and also making data transfer to the client unnecessary. Such solutions include Microsoft's Windows 2003 server (with the corresponding remote desktop protocol (RDP)), Citrix Presentation Server, VNC, or SGI's OpenGL Vizserver. However, most of these solutions do not allow applications to use graphics hardware acceleration. The SGI OpenGL Vizserver did allow hardware accelerated graphics applications to be run over the network: it allocated an InfiniteReality pipeline to an application controlled over the network. However that pipeline could then not be used locally any longer and was also blocked for other users. Thus effectively all that the Vizserver was doing was extending a single workplace to a different location in the network. The same is true for VNC.
For general graphics applications (i.e., not specifically volume rendering applications), such as computer games, solutions have been proposed to combine two graphics cards on a single computer (i.e., the user's computer) in order to increase the rendering performance, specifically NVIDIA's SLI and AMD/ATI's Crossfire products. In these products, both graphics cards receive the exact same stream of commands and duplicate all resources (such as textures). Each of the cards then renders a different portion of the screen—or in another mode one of the cards renders every second image and the other card renders every other image. While such a solution is transparent to the application and therefore convenient for the application developers it is very limited too. Specifically the duplication of all textures effectively eliminates half of the available physical texture memory.
An object of the invention is to provide digital data processing methods and apparatus, and more particularly, by way of example, to provide improved such methods and apparatus for visualization of image data.
A further object of the invention is to provide methods and apparatus for rendering images.
A still further object of the invention is to provide such methods and apparatus for rendering images as have improved real-time response to a user's interaction.
Yet a still further object of the invention is to provide such methods and apparatus as allow users to interactively explore the rendered images.